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ABSTRACT 

This  paper  describes  the  ongoing  research  on  3-dimensional  (3-D)  imaging  at  FOI.  Specifically,  we 
address  the  new  possibilities  brought  by  laser  radars,  focusing  on  systems  for  high  resolution  3-D 
imaging.  3-D  laser  radar  is  a  viable  technology  in  the  effort  to  prevent  and  combat  crime  and  terrorism. 
Real  time  3-D  sensing  is  a  reality  and  can,  besides  more  conventional  techniques  such  as  stereo  vision 
and  structured  light,  be  achieved  by  range  imaging.  Current  development  of  3-D  sensing  flash  imaging 
laser  radars  will  provide  the  capability  of  high  resolution  3-D  imaging  at  long  ranges  with  cm-resolution 
at  full  video  rate.  In  all  probability,  this  will  revolutionise  many  applications,  including  law  enforcement 
and  forensic  investigations.  In  contrast  to  conventional  passive  imaging  systems,  such  as  CCD  and 
infrared  (IR)  techniques,  laser  radar  provides  both  intensity  and  range  information  and  has  the  ability  to 
penetrate  certain  scene  elements  such  as  vegetation  and  windows.  This,  in  turn,  means  new  potentials,  for 
example,  in  object  recognition  and  identification  and  we  address  some  of  these  new  capabilities  of  3-D 
laser  radar  systems.  The  results  clearly  show  that  3-D  imaging  laser  radar  systems  are  useful  in  a  variety 
of  situations  that  can  be  used  in  the  criminal  justice  system  today  to  enable  technologies  for  preventing 
and  combating  crime  and  terrorism. 


1.0  INTRODUCTION 

With  3-D  laser  radar  a  new  dimension  is  added  to  active  imaging.  In  addition  to  intensity  and  angular 
coordinates,  also  range  is  included  in  the  image.  Range  resolved  imaging  gives  the  ability  to  recognize  and 
identify  objects  and  people  based  on  3-D  geometrical  shape,  as  well  as  other  parameters  such  as  reflection 
and  contrast.  It  also  adds  the  ability  to  “see”  what  is  hidden  behind  partly  obscuring  objects,  such  as 
vegetation,  camouflage  nets,  curtains,  and  Venetian  blinds.  This  ability  can  be  used  in  surveillance,  and 
for  reconnaissance  and  recognition.  The  high  resolution  and  accuracy  of  3-D  laser  radar  makes  it  possible 
to  make  accurate  measurements  in  a  3-D  image,  e.g.,  to  identify  a  person  or  to  make  a  complete  3-D 
model  of  the  scene  of  a  crime. 

In  this  paper  we  present  the  technique  that  is  used  for  active  imaging  with  laser  radar  and  give  examples  of 
how  this  technology  can  be  of  use  in  the  work  to  combat  crime  and  terrorism.  In  section  2  we  describe 
some  of  the  laser  radar  techniques  that  can  be  used  for  3-D  imaging.  We  give  examples  of  some 
applications  of  3-D  imaging  systems  in  Section  3,  and  in  Section  4  we  present  some  ongoing  work 
concerning  the  usage  of  laser  radar  systems,  signal  and  image  processing  for  recognition,  visualization, 
and  modelling. 
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2.0  3-D  IMAGING  LASER  RADAR 

There  are  several  principles  for  3-D  imaging  laser  radar.  Basically,  a  pulsed  laser  is  used  for  illumination 
and  a  detector  collects  the  reflected  radiation  from  the  scene.  The  active  illumination  with  a  laser  results  in 
complete  independence  of  ambient  light  conditions  (such  as  day  or  night),  and  hence  the  image  contrast  is 
very  robust  in  that  respect. 

If  an  illuminating  laser  pulse  hits  some  partly  transmitting  material,  or  some  concealing  object  with  gaps 
in,  and  is  reflected  from  a  target  surface  behind  the  concealment,  this  can  be  detected  by  the  sensor  as 
depicted  in  Figure  1.  One  can  think  of  many  applications  where  it  is  of  interest  to  observe  objects  and 
areas  that  are  more  or  less  obscured  by,  e.g.,  vegetation,  camouflage  nets,  curtains,  Venetian  blinds,  or 
shaded  windows.  Visible  and  near  infrared  (NIR)  wavelengths  penetrate  windows  in  buildings  and 
vehicles.  This  can  be  used  for  reconnaissance  purposes  to  detect  if  humans  are  present  in  a  building  or  a 
vehicle.  Other  applications  are  terrain  mapping,  where,  e.g.,  trees,  roads,  and  buildings  can  be  classifled. 
Object  recognition  is  facilitated  by  3-D  information  in  comparison  to  a  2-D  image,  but  also  more  complex. 


Figure  1.  The  illuminating  laser  pulse  first  hits  the  canopy  of  the  tree, 
which  results  in  the  first  echo.  Part  of  the  laser  pulse  penetrates 
the  canopy  and  is  finally  reflected  by  the  stem,  giving  the  last  echo. 

In  the  last  few  years  there  has  been  a  fast  development  of  new  sensors  and  sensor  capabilities.  3-D 
imaging  laser  radars  have  generally  been  scanning  systems,  usually  with  mechanical  scanners.  Hence,  data 
collection  has  been  quite  time  consuming.  Furthermore,  capturing  moving  objects  has  not  been  possible.  A 
new  generation  staring  systems  (flash  laser  radar)  is  now  being  developed,  where  a  complete  rangefinder 
in  included  in  each  pixel  in  the  form  of  a  time-resolved  receiver  and  read-out  integrated  circuit  (ROIC). 
This  enables  the  capture  of  a  complete  3-D  image  with  just  one  transmitted  laser  pulse. 

2.1  Gated  viewing  or  burst  illumination 

With  the  technique  of  gated  viewing  (GV),  also  called  burst  illumination,  the  sensor  can  be  a  simple 
camera  constructed  for  the  laser  wavelength,  but  the  shutter  is  connected  to  and  synchronized  with  a  short 
illuminating  pulsed  laser.  With  an  adjustable  a  delay  setting,  corresponding  to  the  time-of-flight  back  and 
forth  to  a  desired  range,  the  opening  of  the  camera  shutter  is  controlled.  This  exposes  the  camera  only  for  a 
desired  range  slice,  with  the  slice  as  deep  as  the  laser  pulse  length  plus  the  shutter  open  time.  The  delay 
can  be  changed  through  a  predefined  program,  resulting  in  a  number  of  slices  representing  different 
ranges,  i.e.,  a  3-D  volume.  The  set-back  of  this  system  is  the  power  inefficiency,  since  every  range  slice 
image  requires  a  total  scene  illumination.  The  advantage  is  the  low  cost  and  robustness,  since  rather 
simple  components  can  be  used.  As  active  illumination  is  used,  GV  systems  have  night  capability.  With 
gated  viewing,  a  number  of  advantages  over  passive  imaging  can  be  listed: 

•  Suppression  of  reflexes  from  atmosphere,  smoke,  and  terrain. 


16-2 


RTO-MP-SCI-158 


UNCLASSIFIED/UNLIMITED 


UNCLASSIFIED/UNLIMITED 

3-D  Imaging  by  Laser  Radar  and  Applications  in 
Preventing  and  Combating  Crime  and  Terrorism 


•  Potential  for  detection  of  optics  (e.g.,  optical  sensors  or  binoculars)  at  long  ranges. 

•  Target  range  can  be  obtained  with  image  information 

•  Silhouette  detection 

•  Suppression  of  optically  annoying  backgrounds  (e.g.,  direct  light)  by  short  exposure  time  and 
active  illumination 

•  Potential  for  better  weather  performance  compared  to  passive  electro-optical  systems,  and 
possibility  to  “see”  through  smoke  and  haze 

Furthermore,  it  has  been  shown  that  a  rather  small  set  of  gated  images  can  give  high  resolution  3-D 
images.  By  taking  the  depth  information  into  account,  the  3-D  volume  of  an  object  can  be  constructed  by  a 
few  gated  images. 

2.2  Laser  scanner 

A  straight-forward  method  to  acquire  3-D  information  about  a  scene  is  to  scan  the  object  with  a  single 
element  detector  laser  radar.  With  every  laser  pulse  a  very  small  part  of  the  object  is  illuminated  and  the 
time-of-flight  is  stored.  Some  detectors  give  a  time-resolved  pulse  response  (full  waveform),  whereas 
other  detectors  only  give  the  time  for  the  pulse  return  (above  a  certain  threshold).  With  some  systems  it  is 
possible  to  store  first  and  last  echo,  or  even  more  returns,  each  echo  representing  a  different  target  range. 
An  advantage  of  a  scanning  system  is  the  possibility  to  achieve  high  angular  resolution.  The  main 
disadvantage  is  the  long  data  acquisition  time,  which  prevents  the  capture  of  moving  objects.  Scanning 
laser  radar  systems  can  be  ground  based  laser,  see  Figure  2  [1,  2],  or  airborne,  see  Figure  3  [3].  Ground 
based  systems  can  be  used  for  3-D  modelling  and  recognition  of  small  objects,  whereas  airborne  systems 
are  mainly  used  for  terrain  mapping. 


Figure  2.  Two  ground  based  scanners  that  are  used  at  FOI. 
Optech  ILRIS  3-D  (left)  and  Riegl  LMS-Z210  (right). 
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Figure  3.  TopEye  is  an  airborne  iaser  radar  system  for  air-to-ground  iaser  scanning.  (From  [3]) 


2.3  3-D  flash  laser  radar 

The  development  of  focal  plane  array  (FPA)  detectors  with  timing  capability  in  each  pixel  has  made  non¬ 
scanning  3-D  imaging  laser  radars  feasible.  FPA:s  with  up  to  128x128  elements  and  read-out  integrated 
circuits  (ROIC)  are  available  today [4].  With  such  a  system,  the  frame  rate  can  be  increased  to  video  rate 
(50  Hz  or  60  Hz),  to  enable  also  the  capture  of  moving  objects.  The  sensor  is  not  larger  than  an  ordinary 
camera,  se  Figure  4,  excluding  the  laser  source,  which  can  be  fit  to  the  application,  and  the  data 
acquisition  platform,  normally  a  computer.  In  short  range  system  versions  the  laser  can  be  incorporated 
into  the  camera  unit  itself  An  image  example  is  shown  in  Figure  5.  For  each  laser  pulse  the  complete 
waveform  is  acquired  in  each  pixel,  and  from  this  data,  the  range  information  can  be  calculated,  as  seen  in 
Figure  6. 


Figure  4.  The  flash  3-D  sensor,  excluding  the  illuminating  laser. 
The  scale  is  in  inches.  (Courtesy:  R.  Richmond,  AFRL,  USA) 
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Figure  5.  Example  data  from  a  flash  3-D  laser  radar  (Courtesy:  R.  Richmond,  AFRL,  USA). 
Left:  A  photograph  of  a  truck.  Right:  One  range  slice  from  a  3-D  image. 

The  image  is  slightly  processed  at  FOI. 


Figure  6.  The  extracted  depth  from  one  laser  pulse. 


3.0  APPLICATIONS 

3-D  imaging  with  laser  radar  is  a  fairly  new  application,  and  has  become  commonly  used,  e.g.,  for  terrain 
mapping  and  data  collection  for  3-D  modelling.  Some  additional  applications  for  3-D  laser  radar  are  listed 
below: 


•  Reflectance  measurements.  From  measurements,  target  reflectance  can  be  calculated.  Targets  are 
usually  military  vehicles  or  backgrounds. 

•  3-D  modelling.  3-D  models  are  built  from  data  sets  and  are  used  for  further  simulations  in 
different  kinds  of  applications.  Models  of  both  vehicles  and  backgrounds  can  be  built. 
Backgrounds  can  be  coordinated  with  GPS  data. 

•  Multiple  views.  Data  sets  (views)  from  different  locations  can  be  stitched  together,  to  increase  the 
point  density  in  the  3-D  point  cloud,  and  to  improve  the  ability  to  uncover  partly  hidden  objects. 

•  Classification  and  recognition.  Algorithms  for  classification  and  recognition  are  developed. 
Detected  points  can,  for  instance,  be  compared  to  3-D  models  and/or  surfaces  (e.g.  camouflage 
nets).  3-D  data  sets  can  also  be  used  in  algorithms  for  body  shape  and  motion  pattern  recognition. 

3.1  Penetration 

With  3-D  laser  radar  technology  it  is  possible  to  “see  through”  concealing  objects,  but  of  course  there 
needs  to  be  holes  where  at  least  part  of  the  illuminating  laser  beam  has  a  free  line-of-sight  from  the  sensor 
to  the  object  that  is  going  to  be  imaged.  Using  the  last  pulse  mode  of  a  laser  scanner  it  is  possible  to 
penetrate  for  instance  a  forest  [5]. 
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For  urban  missions,  a  UGV  can  be  used  for  “window  clearing”  and  can  locate  people  in  dark  rooms 
behind  Venetian  blinds,  see  Figure  7.  While  passive  EO  sensors  have  difficulties  here,  the  3-D  laser  radar 
is  ideal  looking  for  manlike  “objects”  in  the  room  or  detecting  rifle  optics  etc. 


Figure  7.  Urban  mission  doing  “window  cieaning”  and  iocating  peopie  inside  the  buiidings. 
(image  modified  after  originai  found  at  www.ari.army.mii/wrmd). 


3.2  Urban  modelling 

Airborne  laser  scanner  data  has  shown  to  be  suitable  for  creating  high  resolution  3-D  virtual  environment 
models.  Such  models  can  be  used  for  a  number  of  applications,  e.g.,  crisis  management,  decision  support 
systems,  visual  simulations,  etc.  Much  research  and  work  is  currently  going  on  to  develop  methods  for 
processing  and  extracting  geographic  information  from  airborne  laser  scanner  data.  When  constructing 
environment  models,  a  first  step  is  to  estimate  the  ground  surface.  Different  methods  have  been  developed 
for  this.  In  [6],  eight  different  methods  are  compared.  After  having  extracted  the  ground  surface,  the 
remaining  laser  points  above  ground  can  be  further  separated  into  natural  (vegetation)  or  man-made 
(buildings)  objects  using  measurements  based  on  height  variations  [7,  8].  Natural  objects  will  show  larger 
vertical  variations  than  within  roof  faces  of  buildings,  since  the  laser  beam  can  penetrate  the  canopy  of 
trees.  Using  the  elevation  data  within  each  segment  classified  as  buildings,  3-D  models  of  buildings  can  be 
reconstructed  [9-11].  Areas  classified  as  vegetation  can  be  further  analyzed  to  extract  individual  trees  [12, 
13].  For  the  detected  trees,  different  tree  attributes  such  as  position,  tree  height  and  crown  diameter  can  be 
estimated.  These  extracted  data  sets  of  information  are  suitable  for  construction  of  3-D  virtual 
environment  models. 

3.3  Recognition 

Object  recognition  in  images  has  been  extensively  studied  in  the  computer  vision  community  the  last 
decades.  In  the  general  case,  it  is  an  extremely  hard  problem,  since  it  relies  on  context-dependent 
background  knowledge  and  thus  tightly  interconnected  to  the  artificial  intelligence  problem.  In  certain 
specific  cases,  object  recognition  is  practically  applicable  thanks  to  restrictions  to  a  specific  class  of 
objects,  external  conditions,  and/or  incorporation  of  a  priori  information.  Examples  are  face  recognition, 
modelling  of  buildings,  and  military  target  recognition.  Still,  even  with  these  restrictions,  the  applications 
are  limited.  An  inherent  problem  is  that  the  2-D  projection  (i.e.,  and  image)  of  a  3-D  object  does  not 
contain  much  information  on  the  object’s  3-D  structure,  even  if  much  of  it  can  be  regained  by  different 
“Shape  from  X”  techniques  [14](i.e.,  various  techniques  for  extracting  3-D  information  from  2-D  data). 

Obviously,  a  sensor  acquiring  3-D  information  gives  a  huge  advantage  in  this  respect.  However,  3-D 
object  recognition  is  not  yet  as  well  explored  as  in  2-D,  as  the  sensors  are  not  as  widespread  as  2-D 
imaging  sensors.  Therefore,  the  current  applications  are  somewhat  limited  for  most  people.  This  is  due  to 
change  as  systems  for  capturing  3-D  information  become  less  expensive. 
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Whereas  3D-to-2D  correspondence,  Shape  from  X,  and  illumination  (including  shadows  and  reflectance) 
all  are  important  problems  in  2-D  object  recognition,  other  problems  need  to  be  considered  in  3-D 
recognition.  The  two  most  immediate  are  probably  surface  reconstruction  and  model-to-surface  (or  point 
scatter)  matching.  Recently,  signiflcant  progress  has  been  made,  by  exploiting  new  data  representations. 
The  spin-image  representation  introduced  by  Zhang  and  Hebert  [15]  has  been  exploited  by  Vasile  [16]  to 
achieve  automatic  detection  and  recognition  of  military  targets.  A  spin-image  is  a  representation  of  a  small 
patch  of  an  object’s  surface,  with  the  useful  property  that  it  is  invariant  to  rotation,  i.e.,  regardless  of  from 
which  direction  you  look  at  an  object,  the  spin-images  will  be  the  same.  The  same  spin-image 
representation  has  been  further  developed  by  Ruiz-Correa  et  al.  [17],  combining  spin-images  into 
symbolic  surface  signatures  used  to  learn  shape-classes.  Even  though  their  results  are  impressive, 
application  to  real-world  problems  are  yet  to  be  seen.  However,  the  potential  for  reliable  object 
recognition  (at  least  when  restricted  to  certain  classes,  like  military  targets)  is  huge  and  applicability 
within  reach. 

Even  if  systems  for  automatic  object  recognition  are  not  yet  available  in  cheap  commercial-of-the-shelf 
(COTS)  products,  there  are  situations  where  manual  interaction  is  acceptable.  Consider  the  following 
scenario,  successfully  applied  in  facial  identification  at  the  Netherlands  Forensic  Institute.  An  unknown 
person  A  is  observed  by  a  surveillance  camera,  and  the  problem  is  to  determine  if  person  A  is  identical  to 
an  (available)  person  B.  A  traditional  approach  is  to  capture  an  image  of  person  B,  where  the  imaging 
conditions  (illumination,  camera  position,  head  pose)  are  as  similar  as  possible  to  the  conditions  when 
capturing  the  image  of  person  A.  The  two  images  are  then  (manually  but  computer-aided)  compared,  the 
result  often  being  a  matter  of  opinion. 

Instead,  assume  that  we  make  a  3-D  scan  of  person  B’s  head  and  create  a  3-D  model.  We  identify  a  small 
set  of  points  (maybe  four)  on  the  head  of  person  A  and  mark  the  same  points  on  the  3-D  model  of  person 
B.  We  can  then  visualize  the  3-D  model  in  the  same  frame  as  the  surveillance  image,  taking  into  account 
the  camera  parameters,  and  try  to  align  the  3-D  model  to  the  image.  In  almost  all  cases  where  the  persons 
A  and  B  are  different,  the  alignment  will  fail  due  to  differences  in  the  3-D  structure  of  the  two  heads. 
Naturally,  a  good  match  does  not,  with  the  currently  available  a  priori  information,  imply  that  the  two 
persons  are  identical.  Nevertheless,  the  capability  of  excluding  people  from  a  list  of  suspects  is  also 
important  in  a  forensic  context.  This  technique  is  also  used  in  the  area  of  crano-facial  identification. 


4.0  EXAMPLES 

In  this  section  we  give  describe  in  more  detail  some  of  the  relevant  work  that  is  being  done  at  FOI.  Images 
of  object  hidden  behind  vegetation  and  other  kinds  of  camouflage  are  presented.  Also,  some  examples  of 
urban  modelling,  as  well  as  examples  of  visualization  and  recognition  are  given. 

4.1  Penetration  through  vegetation  and  camouflage 

As  mentioned  above,  the  ability  to  record  the  last  echo  with  a  3-D  laser  radar,  makes  is  possible  to  reveal 
objects  that  are  partly  concealed.  Experimental  data  collected  with  scanning  3-D  laser  radar  systems  shows 
that  an  illuminating  laser  pulse  can  penetrate  through  several  tens  of  meters  into  vegetation,  and  enable 
imaging  of  objects  that  are,  e.g.,  behind  a  forest  line  [5]. 

The  capability  to  “see  through”  vegetation  can  be  enhanced  by  imaging  the  scene  from  several  viewing 
angles  and  then  stitching  the  views  together.  This  is  illustrated  by  the  scenario  in  Figure  8,  where  eight 
views  of  a  fairly  dense  group  of  trees  were  recorded  from  various  viewing  angles.  A  Volvo  V40  is  placed 
behind  the  trees.  In  Figure  9  the  combined  point  cloud  is  rotated,  revealing  the  vehicle  that  is  hidden 
behind  the  trees. 
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Figure  8.  In  the  middle  a  3-D  laser  radar  image  of  a  group  of  trees  stitched  together  from  eight 
different  views.  Around  is  a  series  of  visual  views  taken  from  each  individual  imaging  position 
of  the  laser  scanner.  The  red  rectangles  indicate  the  location  of  the  hidden  car. 


Figure  9.  3-D  laser  radar  images  from  different  viewing  angles  can  be  combined  to 
give  a  well-resolved  image  of  the  target.  This  image,  combined  from  eight  views, 
is  re-projected  and  shows  a  Volvo  V40  hidden  behind  dense  vegetation. 


Recent  field  tests  conducted  by  FOI  show  that  3-D  laser  radar  can  penetrate  camouflage  nets  and  provide 
an  image  of  concealed  objects.  An  example  is  shown  in  Figure  10,  where  a  man  is  standing  under  a 
camouflage  shelter.  In  the  3-D  laser  radar  image,  with  last  echo  setting,  the  man  is  clearly  visible.  The  low 
reflectance  of  the  face  causes  the  man’s  head  to  disappear. 
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Figure  10.  A  photograph  of  a  vehicle  camouflage  shelter  to  the  left.  To  the  right,  a  re-projected 
3-D  laser  radar  image  (last  echo)  of  the  scene,  also  revealing  a  man  standing  in  the  middle 
of  the  shelter.  Note  the  low  reflection  from  the  face.  As  the  last  echo  is  recorded,  the 
pulse  returns  from  the  man  under  the  net  cause  a  “shadow”  on  the  front  of  the  net. 


Figure  11.  Top  left  shows  a  photo  of  a  window  with  the  Venetian  blinds  down 
(approx.  3  mm  spacing),  and  top  right  shows  an  IR  image  of  the  same  window. 

Bottom  left  shows  a  3-D  laser  radar  image  (rotated  45  degrees  with  respect  to  the 
viewing  angle)  showing  a  man  inside  the  room.  Bottom  right  show  a  section  of 
the3-D  data  point  cloud  (front  view).  The  person  is  highlighted  in  red  for  clarity. 


The  ability  to  image  objects  that  are  partly  concealed,  e.g.,  by  Venetian  blinds,  has  been  studied  at  FOL 
Whereas  thermal  imagers  can’t  be  used  for  imaging  through  windows,  due  to  the  wavelength  dependent 
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transmission  of  the  glass,  the  laser  radar  illumination  at  1.5  gm  wavelength  is  transmitted  through  the 
window  and  can  be  used  to  image  objects  inside  a  building.  This  is  illustrated  in  Figure  11,  where  a  person 
is  standing  in  a  room  with  the  Venetian  blinds  covering  the  window.  Another  example  is  shown  in  Figure 
12  with  an  image  from  a  flash  3-D  laser  radar. 


Figure  12.  To  the  left  a  photograph  of  a  scene  with  a  man  concealed  behind  Venetian  blinds. 
A  reflection  of  the  camera  flash  can  be  seen  in  a  reflector  (placed  on  a  stool  next  to  the  man). 
The  flash  3-D  laser  radar  image  to  the  right  reveals  the  man  behind  the  blinds. 
(Courtesy:  R.  Richmond,  AFRL,  USA) 


4.1.1  Reflection  from  human  faces  and  clothes  at  1.5  pm 

The  reflection  from  human  faces  at  the  eye-safe  wavelength  1.5  pm  is  obviously  of  interest  for  applications 
like  detection  of  people  in  concealed  environments  and  face  recognition.  An  example  from  some  initial 
experiments  is  shown  in  Figure  13  [5].  Bare  skin  seems  to  have  lower  reflectance  than  hair  and  clothes. 


Figure  13.  To  the  left  a  photograph  and  to  the  right  a  laser  radar  image  (reflectance)  at 
1.5  pm  wavelength.  Note  the  high  reflection  from  the  clothes  and  hair  compared  with 
the  reflection  from  the  skin.  Also  note  that  the  eyes  were  shut  during  the  scan. 
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4.2  Urban  modelling 

Modelling  of  urban  environments  and  specific  scene  elements  is  highly  facilitated  by  3-D  data  from  a  laser 
radar.  An  example  of  urban  modelling  is  shown  in  Figure  14,  which  depicts  the  market  place  in  “Old 
Linkoping”.  The  data  was  taken  with  one  scan  with  a  3-D  laser  radar  positioned  at  the  centre  of  the  market 
place. 


Figure  14.  Data  from  one  scan  with  a  3-D  iaser  radar, 
showing  the  market  piace  in  “Oid  Linkoping”. 


4.2.1  Recognition  and  classifications 

Methods  have  been  developed  for  processing  airborne  laser  scanner  data.  The  long  term  goal  is  to  develop 
methods  to  automatically  extract  geographic  information  for  supporting  the  construction  of  3-D  virtual 
environment  models.  The  methods  work  on  gridded  data.  First,  the  irregularly  distributed  laser  points  are 
resampled  into  regular  grids  (0.25x0.25m^).  Two  grids  are  stored:  one  grid  containing  the  highest 
elevation  value  in  each  cell  (DSMmax)  as  shown  in  Figure  15,  upper  left,  and  one  grid  containing  the  lowest 
value  (DiSMmin). 

The  first  class  to  be  determined  is  the  ground  surface.  The  method  to  estimate  the  ground  surface  (DTM) 
works  on  DSM^mn  and  is  based  on  active  contours  [18].  The  method  is  based  on  the  theory  of  active  shape 
models  [19,  20].  The  active  contour  can  be  seen  as  a  net  being  pushed  upwards  from  underneath  the 
surface  and  attached  the  laser  points. 

The  data  points  above  the  ground  surface  are  then  further  classified.  Classification  of  buildings  starts  by  a 
segmentation  using  multiple  return  information.  Multiple  returns  occur  when  different  parts  of  a  laser 
pulse  “footprint”  are  reflected  from  different  elevations.  For  example,  at  edges  of  buildings  and  within 
vegetation,  the  laser  footprint  is  often  split  so  one  part  is  reflected  from  the  top  of  the  object  and  the  other 
part  is  reflected  from  ground,  see  the  middle  image  in  Figure  15.  Using  this  information  of  multiple 
returns,  much  of  the  vegetation  can  be  removed  or  at  least  separated  from  buildings.  Separate  segments  are 
defined  for  each  group  of  pixels  that  are  more  than  2  meters  above  the  ground  surface.  Each  segment  is 
then  classified  as  building  or  non-building  based  on  measures  of  curvature,  maximum  slope  [21]  and 
shape  using  an  artificial  neural  network.  Next,  posts  (e.g.,  lamp  posts)  are  classified  by  detecting  narrow, 
high  objects.  Remaining  pixels  which  are  more  than  2  meters  above  the  ground  surface  are  classified  as 
vegetation.  Pixels  classified  as  ground  are  further  classified  as  “road”  (roads,  paths,  other  paved  areas,  etc) 
or  “non-road”  using  the  intensity  of  the  returned  pulses  as  shown  in  Figure  15,  upper  right. 
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The  elevation  data  within  each  segment  classified  as  buildings  are  used  to  construct  3-D  models  of 
buildings.  First,  planar  roof  faces  are  extracted  using  a  clustering  of  surface  normals.  Next,  topological 
points  are  inserted  where  each  face’s  neighbour  changes.  Sections  between  these  points  separating  the  roof 
faces  are  defined  as  intersections,  edges  or  both.  In  order  to  obtain  building  models  with  piecewise  linear 
walls,  straight  lines  are  estimated  along  edge  sections  using  the  2-D  Hough  transform.  New  points  are 
inserted  at  the  intersections  of  the  estimated  lines.  Using  this  information,  3-D  models  of  building  are 
constructed. 

Finally,  single  trees  are  extracted  in  the  areas  classified  as  vegetation  [22].  For  each  extracted  tree,  the 
position,  tree  height  and  crown  diameter  are  estimated  using  the  elevation  data  and  the  area  of  the  crown 
segment.  The  different  layers  of  extracted  information  can  be  used  to  construct  a  3-D  virtual  environment 
model  as  shown  in  Figure  16. 


Figure  15.  Left:  Gridded  height  data  (DSM^ax)-  Middie:  Muitipie  return  image.  Right:  Ciassification 
resuits,  buiidings  (biack),  vegetation  (gray),  roads  (dark  gray),  posts  (white). 


Figure  16.  3-D  virtuai  environment  modei,  showing 
the  ground  surface  with  buiiding  and  tree  modeis. 


4.2.2  Visualitation  and  surveillance 

The  availability  of  3-D  models  of  an  urban  area  does  not  only  allow  visualisation  of  the  modelled  objects 
and  buildings  themselves  but  also  of  sensor  data.  This  can  drastically  improve  situation  awareness  in 
situations  like  surveillance,  crisis  management,  and  urban  fighting  situations. 
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The  most  common  way  of  analyzing  data  from  (multiple)  surveillance  cameras  is  basically  to  sit  and 
watch  the  monitors.  One  of  the  problems  with  such  an  approach  is  that  although  it  is  easy  to  notice  when 
an  extraordinary  event  occurs,  it  is  still  difficult  to  perceive  what  is  actually  happening.  Imagine  some 
form  of  urban  crisis,  e.g.,  riots,  panic,  or  fighting.  People  (and  groups  of  people)  and  vehicles  move  into 
and  out  of  the  fields-of-view  of  the  cameras.  It  is  not  an  easy  task  for  a  manual  operator  to  connect  all 
these  micro-events  and  see  the  larger  picture. 

A  solution  is  to  visualize  the  3-D  model  of  the  area,  and  project  the  recorded  video  on  the  model.  In  the 
synthetic  world,  the  cameras  are  replaced  by  video  projectors,  projecting  what  the  cameras  in  the  real 
world  sees  [23].  There  are  several  advantages  of  this  approach: 

•  It  gives  an  overview  of  the  area  and  shows  the  various  camera  recordings  in  the  same  view  at  the 
same  time. 

•  It  becomes  obvious  how  the  different  recordings  are  related  in  time  and  space. 

•  Other  sensor  data,  like  intruder  detectors,  acoustic  or  seismic  tracking  and  detection,  can  easily  be 
integrated  in  the  same  view.  For  example,  if  a  threat  is  detected  outside  the  field  of  view  of  all 
cameras,  it  can  still  be  marked  in  the  synthetic  3-D  environment  and  directly  related  to  visual 
(model  and  video)  data.  Moreover,  assume  that  a  shooter  is  located  using  an  acoustic  sensor 
network  [24],  not  only  the  shooter  can  immediately  be  marked  in  the  3-D  model  but  also  the  entire 
area  within  his  field  of  view,  i.e.,  the  current  risk  area. 

4.3  Recognition 

4.3.1  Recognition  of  humans,  human  activity  and  objects 

Computer-based  image  processing  can  on  a  high  level  be  divided  into  two  groups;  image  enhancement  and 
automatic  analysis.  The  purpose  of  both  approaches  is  to  clarify  the  contents  in  the  image(s)  to  support  an 
operator.  In  image  enhancement  the  over-all  intensity  (or  range  data)  in  the  image  is  improved  but  the 
decisions  of  what  parts  of  the  image  that  are  interesting,  and  what  they  contain,  need  to  be  made  by  the 
operator.  In  some  cases  this  is  a  satisfactory  solution.  In  other  cases,  for  example  when  large  image  (or  3- 
D  data)  sets  have  to  be  analyzed  in  a  short  time  frame,  automation  of  parts  of  the  processing  can  support 
the  operator.  Examples  are  cueing  to  interesting  parts  of  the  image  and  selection  of  image  frames  (or  range 
frames)  where  interesting  objects  are  present.  Furthermore,  post  processing  of  data  can  support  the 
operator  in  interpreting  images  of  partly  occluded  objects,  for  example  objects  or  humans  that  are  located 
behind  camouflage  nets,  terrain  foliage,  Venetian  blinds  or  glass  in  buildings  and  vehicles. 

Image  enhancement  of  gated  viewing  images  has  been  shown  in  [25].  In  gated  viewing  the  images  are 
subject  to  noise  and  disturbance  mainly  originating  from  target  reflections,  atmospheric  disturbances 
(turbulence,  speckle  noise),  camera  and  laser  performance.  Further,  the  platform  may  be  in  motion.  By 
tracking  the  translation  and  rotation  of  sharp  lines  and  edges  in  the  image  during  the  turbulence  and 
platform/camera  motions,  it  is  possible  to  apply  temporal  smoothing  to  reduce  the  effects  of  (random) 
illumination  variances.  Using  this  kind  of  image  enhancement  human  activities  at  km-distances,  e.g., 
people  in  a  vehicle,  can  be  enhanced  and  possible  to  detect  and  interpret  by  an  operator,  see  Figure  17. 

Laser  illumination  causes  intense  reflections  in  optics,  for  example  in  headlight  reflectors  and  other 
reflective  materials  on  vehicles.  This  has  been  used  to  detect  vehicles  hidden  behind  camouflage  nets  and 
terrain  foliage  [26].  In  this  work,  “hot  spots”  originating  from  highly  reflective  areas  on  the  target  were 
detected  using  a  statistical  approach. 

Some  short  range  GV  systems  have  enough  resolution  to  resolve  a  human  face  for  visual  recognition. 
There  are  also  biometric  measures  that  can  be  used  for  recognition  of  motion  patterns  [25].  Biometrical 
measures  discussed  are  the  15  main  parts  of  the  human  body  (head,  neck,  torso,  upper  arm,  forearm,  hand. 
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hip,  calf,  foot).  By  tracking  the  body  parts,  and  their  interrelations,  patterns  that  are  specific  for  each 
individual  can  be  extracted. 


Figure  17.  Image  from  a  gated  viewing  system  at  night,  showing  two  soldiers  in  the  front 
seat  of  a  command  vehicle,  both  holding  up  their  arms  to  cover  a  lamp.  Notice  the 
reflection  in  the  headlights  and  the  clearly  visible  antenna  on  the  roof. 

For  comparison,  a  similar  vehicle  is  shown  in  the  right  photograph. 

In  military  applications,  laser  radar  have  been  used  for  some  decades  for  semiautomatic  or  automatic 
detection  and  recognition  of  military  targets,  like  vehicles,  see  [27]  for  a  recent  overview.  Traditionally,  3- 
D  laser  radar  have  been  used,  but  since  a  sequence  of  GV  images  can  be  transformed  to  a  3-D  volume 
[28],  some  algorithms  from  military  applications  can  be  used  for  this  kind  of  data  as  well. 

4.3.2  3-D  imaging  using  range  gated  viewing 

With  appropriate  gate  control,  we  can  use  multiple  images  from  a  gated  viewing  system  for  making  a  3-D 
reconstruction  of  an  imaged  scene.  The  first  algorithm  for  this  purpose  developed  at  FOI  [29]  was  applied 
to  data  acquired  with  an  experimental  long-range  gated  viewing  system  at  a  range  of  7  km  from  the  target 
scene.  Developments  of  the  original  algorithm  have  lately  provided  significantly  increased  efficiency  in 
data  collection  requirements  and  improved  range  resolution.  A  related  technique  for  high  accuracy  3-D 
reconstruction  at  shorter  ranges  has  recently  been  reported  by  Busck  et  al.  [30]. 

4.3.2. 1  Basic  Principles  of  3-D  Reconstruction 

Successively  moving  the  gate  over  a  scene  in  steps  along  the  range  axis  produces  an  intensity  variation  in 
each  individual  image  pixel.  This  intensity  profile  provides  the  basis  for  a  3-D  reconstruction  from  gated 
viewing.  In  principle,  we  can  measure  the  range  value  of  an  image  pixel  by  determining  at  what  gate  range 
setting  it  lights  up  or  turns  dark  as  the  gate  range  is  successively  shifted  onto  or  off  the  target  scene. 
Separate  treatment  of  each  pixel  is  necessary  for  accurate  reconstruction,  due  to  the  non-uniform 
properties  of  the  laser  beam,  target  scene,  and  atmosphere  when  regarded  over  the  image  surface.  Figure 
1 8  shows  an  example  of  images  from  a  sequence  collected  when  sliding  the  gate  over  a  scene  containing  a 
military  vehicle. 

When  collecting  successive  images  while  shifting  the  range  gate  offset,  the  resulting  pixel  intensity  values 
for  any  given  delay  time  is  proportional  to  the  convolution  of  the  camera  gain  and  the  mirrored  return  laser 
pulse  shape.  Hence  any  measured  pixel  intensity  value  will  be  a  noisy  sample  from  this  curve.  The 
displacement  of  the  curve  along  the  time  axis  corresponds  to  the  range  to  the  target,  so  the  problem  of 
range  imaging  becomes  one  of  estimating  properties  of  this  function  for  every  pixel  from  a  set  of  noisy 
measurements,  using  signal  processing  methods. 
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Figure  18.  Left:  A  visual  photograph  of  the  scene  used  for  gated  viewing  experiments. 
Right:  Gated  viewing  images,  captured  using  four  different  settings  of  the  gate  delay 
time.  This  results  in  the  laser  reflections  from  the  front,  entire,  back,  and  background 
of  the  vehicle  being  imaged  in  the  four  frames.  The  range  to  the  target  is  0.9  km 
and  the  gate  width  40  ns,  corresponding  to  a  physical  depth  of  about  6  m. 


An  example  of  a  3-D  imaging  result  is  shown  in  Figure  19,  where  the  algorithm  has  been  applied  to  the  set 
of  images  previously  presented  in  Figure  18.  The  strength  of  gated  viewing  for  3-D  reconstruction 
compared  to  other  present  techniques  is  the  combination  of  high  lateral  and  range  resolutions  as  well  as 
long  range  capability. 


Figure  19.  Two  different  views  of  a  military  vehicle  reconstructed  from  gated  viewing  measurements. 
Left:  Range  coded  colour  scale  as  seen  from  the  point  of  observation.  Right:  Height  coded 
colour  scale  with  the  3-D  reconstruction  rotated  into  a  different  viewing  angle. 


5.0  CONCLUSIONS 

We  have  described  the  technology  and  potential  use  of  3-D  laser  radar  systems.  With  this  active  sensor  it 
is  possible  to  accurately  and  remotely  measure  range  and  3-D  shape  of  objects,  such  as  vehicles  and 
humans.  A  full  3-D  scan  can  be  used  to  accurately  model  a  scene,  whether  it  is  an  urban  scene,  the  scene 
of  a  crime,  or  individual  objects  (vehicles,  people,  etc.).  Furthermore,  with  3-D  laser  radar  it  is  possible  to 
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penetrate  for  instance  vegetation,  Venetian  blinds,  camouflage  nets,  and  create  an  image  of  what  is  behind. 
This  illustrates  the  strength  of  the  3-D  laser  radar  to  uncover  (partly)  hidden  objects,  and  to  observe 
possible  activity,  e.g.,  through  the  windows  of  a  building. 
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Introduction 


•  With  3-D  laser  radar  (LADAR)  a  new  dimension  is  added 
to  active  imaging 

-  in  addition  to  intensity  and  angular  coordinates,  also  range  is  included 
in  the  image. 

•  Range  resolved  imaging 

-  gives  the  ability  to  recognize  and  identify  objects  and  people  based  on 
3-D  geometrical  shape,  as  well  as  other  parameters  such  as  reflection 
and  contrast. 

-  adds  the  ability  to  “see”  what  is  hidden  behind  partly  obscuring  objects, 
such  as  vegetation,  camouflage  nets,  curtains,  and  Venetian  blinds. 

•  This  ability  can  be  used  in  surveillance,  and  for 
reconnaissance  and  recognition 
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Sensor  equipment 


Gated  viewing  (GV) 

-  Burst  illumination 

-  One  or  several  illuminating 
laser  pulse(s) 

-  Record  one  “range  slice”  of  the 
scene  per  pulse 

3-D  laser  radar 


Simultaneous  recording  of  “all 
ranges”  with  a  single  laser  pulse 

Scanning 

•  Single  element  detector 
Staring  (flash) 

•  Array  detector 
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Range  gate  is 
slided  towards 
longer  ranges: 


Receiver  time  gate 
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3-D  laser  scanners  available  at  FOI 


3-D  flash  laser  radar 


Receiver  Focal  Plane  Array 
(N  X  N  Detectors) 


Laser 
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evolving  technology 


Laser  Flash 
Illuminated  Area 


Receiver 
Single  Pixel  FOV 


•  FPA  (focal  plane  array)  detector, 
128x128  elements 

•  Video  rate 


Courtesy:  AFRL,  USA 


Images  from  a  flash  laser  radar 


Applications  of  3-D  laser  radar 


•  Reflectance  measurements 

-  From  measurements,  target  refleetanee  ean  be  ealeulated.  Targets  are  usually 
military  vehieles  or  baekgrounds. 

•  3-D  modelling 

-  3-D  models  are  built  from  data  sets  and  are  used  for  further  simulations  in 
different  kinds  of  applieations.  Models  of  both  vehieles  and  baekgrounds  ean 
be  built.  Baekgrounds  ean  be  eoordinated  with  GPS  data. 

•  Multiple  views 

-  Data  sets  (views)  from  different  loeations  ean  be  stitehed  together,  to  inerease 
the  point  density  in  the  3-D  point  eloud,  and  to  improve  the  ability  to  uneover 
partly  hidden  objects. 

•  Classification  and  recognition 

-  Algorithms  for  classification  and  recognition  are  developed.  Detected  points 
can,  for  instance,  be  compared  to  3-D  models  and/or  surfaces  (e.g.  camouflage 
nets).  3-D  data  sets  can  also  be  used  in  algorithms  for  body  shape  and  motion 
pattern  recognition. 
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Urban  modelling 
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Reflectance  data 
Multiple  echoes 
Aerial  photographs 


Feature 


Surveillance  system  for  urban  environment 


Classical  video  surveillance:  One  screen  for  each  camera. 
Lack  of  overview,  context,  and  situation  awareness! 


Instead  ... 


1 .  Use  a  3-D  model  of 

the  area  2.  Determine  the  position, 

direction,  and  view  of 

every  camera  3.  Project  the  images 

on  the  3-D  modei 


Example:  Riots  in  urban  environment 
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Acoustic  sensor  network:  Shooting 


Multiple  sensors  in  model  environments 


•  Visualization  of  sensor  data  in  a  3-D  model  gives  context 
and  general  view. 

•  New  sensor  systems  give  new  capabilities: 

-  Localization  and  identification  of  shooters 

-  Identification  of  risk  areas 

-  Tracking  of  people  and  vehicles  between  and  outside  images 

-  Alarm  for  suspicious  behaviours 

•  Gives  tactical  decision  support 


Multiple  views 


Multiple  views  and  re-projection 


What  is  hidden  under  camouflage  a  net? 


•  A  3-D  laser  radar  can  reveal 
objects  under  camouflage  nets 

•  Low  reflectance  from  face. 

•  “Last  echo”  causes  echoes  from 
“hidden”  objects  to  appear  as  a 
“hole”  on  first  surface. 
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Reflection  from  human  faces  and  clothes  at 
1 .5  |jm 


•  Results  from  initial 
experiments 

•  Lower  reflectivity  from 
skin  than  from  clothes 
and  hair 


Gated  viewing 


Some  activity  is 
detected  in  the  vehicle: 
2  people  are  moving 


example 


“Seeing”  through  Venetian  blinds 

Visual  image  3-d  laser  radar 


What  can  be  seen 
through  Venetian  blinds 
in  a  dark  room  (gaps  2-3 
mm)? 


IR  image 


Empty  room 


Person  in  room 


3-D  image  through  window  with  blinds 


Blinds  up 


Blinds  down  (3  mm 


Camouflage  “see-through”  test  with 
flash  ladar  sensor 

Scene  before  blinds  are  lowered 


BLINDS 


DOOR 


TARGET 


STOOL 


CURTAIN 

BOOKCASE 

REFLECTOR 

STOOL 

BOOKCASE 


FOI 


Courtesy:  AFRL,  USA 


Flash  ladar  sees  through 
camouflage  (60  %  obscurity) 


stepping  through  20  samples  captured  with  a  singie  laser  pulse 

DIGITAL  CAMERA  IMAGE 

With  blinds  lowered  128  X  128  FLASH  LADAR  IMAGE 


Courtesy:  AFRL,  USA 


Obscured  Target  Imaging 
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Obscured  target  images 


Open  view 
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3-D  imaging  using  range  gated  viewing 


Sub-range  gate  resolution  is 
possible  with  post  processing 


Conclusions 


•  The  technology  and  potential  use  of  3-D  laser  radar  systems  has 
been  described. 

•  With  this  active  sensor  it  is  possible  to  accurately  and  remotely 
measure  range  and  3-D  shape  of  objects,  such  as  vehicles  and 
humans. 

•  A  full  3-D  scan  can  be  used  to  accurately  model  a  scene,  whether  it 
is  an  urban  scene,  the  scene  of  a  crime,  or  individual  objects 
(vehicles,  people,  etc.). 

•  With  3-D  laser  radar  it  is  possible  to  penetrate,  e.g.,  vegetation, 
Venetian  blinds,  camouflage  nets,  and  create  an  Image  of  what  is 
behind. 

•  This  illustrates  the  strength  of  the  3-D  laser  radar  to  uncover  (partly) 
hidden  objects,  and  to  observe  possible  activity,  e.g.,  through  the 
windows  of  a  building. 
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